Workflow for building and deploying interactive visualizations ¶

Let's say you want to make it easy to explore some dataset, i.e.:

• Make a visualization of the data
• Maybe add some custom widgets to see the effects of some variables
• Then deploy the result as a web app.

You can definitely do that in Python, but you would expect to:

• Spend days of effort to get some initial prototype working in a Jupyter notebook, every time
• Work hard to tame the resulting opaque mishmash of domain-specific, widget, and plotting code
• Start over nearly from scratch whenever you need to:
  • Deploy in a standalone server
  • Visualize different aspects of your data
  • Scale up to larger (>100K) datasets

Step-by-step data-science workflow ¶

Here we'll show a simple, flexible, powerful, step-by-step workflow, explaining which open-source tools solve each of the problems involved:

• Step 1: Get some data
• Step 2: Prototype a plot in a notebook
• Step 3: Model your domain
• Step 4: Get a widget-based UI for free
• Step 5: Link your domain model to your visualization
• Step 6: Widgets now control your interactive plots
• Step 7: Deploy your dashboard

Step 1: Get some data ¶

• Here we'll use a subset of the often-studied NYC Taxi dataset
• About 12 million points of GPS locations from taxis
• Stored in the efficient Parquet format for easy access
• Loaded into a Dask dataframe for multi-core
  (and if needed, out-of-core or distributed) computation

Step 2: Prototype a plot in a notebook ¶

• A text-based representation isn't very useful for big datasets like this, so we need to build a plot
• But we don't want to start a software project, so we use HoloViews:
  • Simple, declarative way to annotate your data for visualization
  • Large library of Elements with associated visual representation
  • Elements combine (lay out or overlay) easily
• And we'll want live interactivity, so we'll use a Bokeh plotting extension
• Result:

Here Points declares an object wrapping df , visualized as a scatterplot of the pickup locations. decimate limits how many points will be sent to the browser so it won't crash.

As you can see, HoloViews makes it simple to pop up a visualization of your data, getting something on screen with only a few characters of typing. But it's not particularly pretty, so let's customize it a bit:

That looks a bit better, but it's still decimating the data nearly beyond recognition, so let's try using Datashader to rasterize it into a fixed-size image to send to the browser:

Ok, that looks good now; there's clearly lots to explore in this dataset. Notice that the aspect ratio changed because Datashader is using every point, including some distant outliers. One way to fix the aspect ratio is to indicate that it's geographic data by overlaying it on a map:

We could add lots more visual elements (laying out additional plots left and right, overlaying annotations, etc.), but let's say that this is our basic visualization we'll want to share.

To sum up what we've done so far, here are the complete 10 lines of code required to generate this geo-located interactive plot of millions of datapoints in Jupyter:

import holoviews as hv, geoviews as gv, dask.dataframe as dd
from colorcet import fire
from holoviews.operation.datashader import datashade
from geoviews.tile_sources import EsriImagery
hv.extension('bokeh')<br>
df = dd.read_parquet('../data/nyc_taxi_wide.parq').persist()
opts = dict(width=700, height=600, xaxis=None, yaxis=None, bgcolor='black')
points = hv.Points(df, ['pickup_x', 'pickup_y'])
taxi_trips = datashade(points, x_sampling=1, y_sampling=1, cmap=fire).options(\*\*opts)
EsriImagery * taxi_trips

Step 3: Model your domain ¶

Now that we've prototyped a nice plot, we could keep editing the code above to explore this data. But at this point we will instead often wish to start sharing our results with people not familiar with programming visualizations in this way.

So the next step: figure out what we want our intended user to be able to change, and declare those variables or parameters with:

• type and range checking
• documentation strings
• default values

The Param library allows declaring Python attributes having these features (and more, such as dynamic values and inheritance), letting you set up a well-defined space for a user (or you!) to explore.

NYC Taxi Parameters ¶

Each Parameter is a normal Python attribute, but with special checks and functions run automatically when getting or setting.

Parameters capture your goals and your knowledge about your domain, declaratively.

Class level parameters ¶

Validation ¶

Instance parameters ¶

Step 4: Get a widget-based UI for free ¶

• Parameters are purely declarative and independent of any widget toolkit, but contain all the information needed to build interactive widgets
• ParamNB generates UIs in Jupyter from Parameters, using ipywidgets

• Declaration of parameters is independent of the UI library used
• ParamBokeh generates UIs from the same Parameters, using Bokeh widgets, either in Jupyter or in Bokeh Server

Step 5: Link your domain model to your visualization ¶

We've now defined the space that's available for exploration, and the next step is to link up the parameter space with the code that specifies the plot:

Note that the NYCTaxiExplorer class is entirely declarative (no widgets), and can be used "by hand" to provide range-checked and type-checked plotting for values from the declared parameter space:

Step 6: Widgets now control your interactive plots ¶

But in practice, why not pop up the widgets to make it fully interactive?

Step 7: Deploy your dashboard ¶

Ok, now you've got something worth sharing, running inside Jupyter. But if you want to share your interactive app with people who don't use Python, you'll now want to run a server with this same code.

• If you used ParamBokeh , deploy with Bokeh Server :
  • Write the above code to a file
  • Add a declaration of which object should be served (see nyc_taxi/main.py below)
  • bokeh serve nyc_taxi/main.py

• If you used ParamNB , deploy with Jupyter Dashboard Server :
  • Use Jupyter Dashboards Extension to select cells from the notebook to display
  • Use preview mode to see layout
  • Use Jupyter Dashboards Server to deploy
  • Note various caveats below

Complete dashboard code ¶

Branching out ¶

The other sections in this tutorial will expand on steps in this workflow, providing more step-by-step instructions for each of the major tasks. These techniques can create much more ambitious apps with very little additional code or effort:

• Adding additional linked or separate subplots of any type;
  see 2 - Annotating your data and 4 - Working with datasets .
• Declaring code that runs for clicking or selecting within the Bokeh plot; see 8 - Custom interactivity .
• Using multiple sets of widgets of many different types; see ParamNB and ParamBokeh .
• Using datasets too big for any one machine, with Dask.Distributed .
• Presenting Jupyter notebooks like this one as slides using RISE .

Future work ¶

• Jupyter Dashboards Server not currently maintained; requires older ipywidgets version
• Bokeh Server is mature and well supported, but does not currently support drag-and-drop layout like Jupyter Dashboards does
• ParamBokeh and ParamNB still need some polishing and work to make them ready for widespread use
• E.g. ParamNB and ParamBokeh should provide more flexible widget layouts
• Lots of work to do!